1. Field of the Invention
The present invention relates to a three-dimensional structure acquisition method, an apparatus and a computer readable medium for generating a realistic image from an original image.
In recent years, three-dimensional computer graphics is in increasing demand in various fields such as games, movies, commercials and the like. However, high labor costs are incurred to create a three-dimensional model for three-dimensional computer graphics. Thus, a method for generating the three-dimensional model automatically or semiautomatically from a real object or a real landscape is required.
2. Description of the Related Art
Conventionally, various inputting methods are proposed for inputting a three-dimensional structure such as an object or a landscape. For example, in a document, R. C. Bolles, H. H. Baker and D. H. Marimont, xe2x80x9cEpipolar-Plane Image Analysis: An Approach to Determining Structure from Motionxe2x80x9d IJCV, vol.1, No.1, 1987, pp.7-55 (which will be referred to as document 1 in the following), a method called epipolar plane image analysis is disclosed. In the epipolar plane image analysis, many images of a landscape taken by a video camera are input, spatio-temporal images are generated, and a three-dimensional structure is obtained by analyzing linear patterns shown in the x-t plane image. This method is simple and easy. But, it is necessary to accurately move the camera with uniform linear motion. When the uniform linear motion is accompanied by unknown motion due to hand shake, vibration and the like, the patterns shown in the x-t plane image do not become linear. Therefore, accuracy for extracting the three-dimensional structure significantly decreases.
In Japanese laid-open patent application No.11-339043 xe2x80x9cMETHOD FOR INPUTTING THREE DIMENSIONAL SHAPE AND RECORDING MEDIUM FOR RECORDING THREE DIMENSIONAL SHAPE INPUT PROGRAMxe2x80x9d (which will be referred to as document 2 in the following), a method for calibrating an image as preprocessing of the method in the document 1 is proposed. In the method, camera fluctuation due to camera shake is estimated from paths of feature points. The estimated camera motion is used for calibrating the image as if the camera keeps uniform linear motion. According to this method, accurate uniform linear motion of the camera becomes unnecessary and the epipolar plane image analysis becomes easy. Similarly, in a method proposed in a document Z. Zhu, G. Xu and X. Lin, xe2x80x9cConstructing 3D Natural Scene from Video Sequences with Vibrated Motions,xe2x80x9d Proc. IEEE VRAIS ""98, 1998, pp. 105-112, camera fluctuation is estimated from optical flow. Then, the image calibrated as if the camera keeps uniform linear motion by using the estimation data.
However, according to the method in the document 2, there is a problem in that accuracy of three-dimensional structure calculation deteriorates when the depth dimension of the three-dimensional structure is large.
In addition, in a method shown in documents, C. Tomasi and T. Kanade, xe2x80x9cShape and Motion from Image Streams : a Factorization Methodxe2x80x94Full Report on the Orthographic Case,xe2x80x9d Computer Science Technical Report, CMU-CS-104, Carnegie Mellon Univ., 1992 (which will be referred to as document 3 in the following) and C. J. Poelman and T. Kanade, xe2x80x9cA Paraperspective factorization method for Shape and Motion Recovery,xe2x80x9d IEEE PAMI, vol. 19, no. 3, 1997, pp. 206-218, three-dimensional coordinates of the image feature points are obtained by inputting a plurality of images of a landscape, determining correspondence of feature points and using a factorization method. This method can be easily performed because a special apparatus or special consideration is unnecessary for inputting an image. However, there is a defect in that error becomes large when the depth dimension of the three-dimensional object is large since the camera model used in the method is not the perspective transform model.
Further, in Japanese laid-open patent application No. 7-146121 xe2x80x9cRECOGNITION METHOD AND DEVICE FOR THREE-DIMENSIONAL POSITION AND ATTITUDE BASED ON VISIONxe2x80x9d, a camera calibration method is disclosed. In the method, a camera position and attitude are estimated from an image of a three-dimensional object taken by the camera, in which the position and the size of the three-dimensional object is known. However, according to this method, a three-dimensional structure of an unknown object or landscape can not be obtained.
In Japanese laid-open patent application No. 8-181903xe2x80x9cIMAGE PICKUP APPARATUSxe2x80x9d, an apparatus is disclosed in which a plurality of images are input, translation and rotation deviations between the images are estimated and a merged image is synthesized. However, according to this method, the three-dimensional structure of an object or a landscape can not be obtained.
In an apparatus disclosed in Japanese laid-open patent application No. 11-183139 xe2x80x9cSECTIONAL AND THREE-DIMENSIONAL SHAPE MEASURING APPARATUSxe2x80x9d, three-dimensional structure of an object is measured from a plurality of images taken by projecting slit light to the object. However, there is a defect that it is expensive to realize the apparatus since a camera needs to operate together with an apparatus for projecting the slit light.
It is an object of the present invention to provide a three-dimensional structure acquisition method, an apparatus and a computer readable medium for acquiring a three-dimensional structure of an object or a landscape accurately even when the depth dimension of the object or the landscape is large.
The above object of the present invention is achieved by a three-dimensional structure acquisition method comprising:
an input step for inputting images from a camera to a computer;
a first extraction step for extracting a three-dimensional structure from the images by a predetermined method;
a generation step for generating depth maps from the three-dimensional structure;
a calibration step for correcting image deviation due to camera fluctuation by using the depth maps;
a second extraction step for extracting a three-dimensional structure from calibrated images by the predetermined method; and
an outputting step for outputting the three-dimensional structure to a storage device.
According to the above-mentioned invention, the three dimensional structure can be acquired accurately since images which are accurately calibrated by using the depth maps are used.
In the three-dimensional structure acquisition method, the predetermined method may be based on epipolar plane image analysis.
According to this invention, the three dimensional structure can be acquired accurately by using the epipolar image analysis.
The three-dimensional structure acquisition method may further include the steps of:
generating synthesized images after extracting the three-dimensional structure;
calculating differences between the synthesized images and the calibrated images;
wherein the generation step, the calibration step and the second extraction step are performed repeatedly until the differences become smaller than or equal to a predetermined value.
According to the above-mentioned invention, accuracy of the three-dimensional structure is further improved. In addition, the process end can be judged by comparing the synthesized images and the calibrated images.
In addition, according to the present invention, a three-dimensional structure acquisition apparatus and a computer readable medium storing a three-dimensional structure acquisition program are provided.